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DECLARATION OF PRIOR INVENTION IN THE U.S. TO OVERCOME CITED 
PATENT OR PXJBLICATION (37 CJFVIL § U31) 

Commissioner for Patents 
P.O. Box 1450 
Alexandria, VA 22313-1450 

Sir; 

Purpose of Declaration 

L This declaration is to establish completion of the invention in this application in the 
United States, at a date prior to January 12, 2001, the effective date of the prior patent 
publication: 

2002/0130955 Al (PeUetier) 

2. The person making this declaration is one of the joint inventors and is the most 
knowledgeable of the joint inventors. 

Facts and Documentary Evidence 

3. To establish the date of completion of the invention of this application, the following 

attached documents are submitted a$ evidence: 

FUJI XEROX (FXPAL) Invention Proposal FXPAL-IP-00-013 dated May 31, 2000 and 
June 1, 2000; 
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Atty. Docket No. CQ10196 
PATENT APPLICATION 

DECLARATION OF PRIOR INVENTION IN THE U.S. TO 
OVERCOME CITED PATENT OR PUBLICATION (37 CFJR. § 1.131) 
US, Application No. 09/981,735 

Fax Cover Sheet from Richard Domingo of Xerox to Steve Roe together with the 
attached FUJI XEROX (FXPAL) Invention Proposal FXPAHP-WMJ13, time-stamped February 
10, 2001 at 13:55. 

4* From these documents, it can be seen that the invention in this application was made: 

at least by the date of May 31, 2000, which is a date earlier than the effective date of (he 
reference. 

Diligence 

5. The attached statement also establishes the diligence of the applicants, from the time of 
their conception, to a time just prior to the date of die reference tip to the filing of this application. 

Time of Presentation of the Declaration 

6. This declaration is submitted after final rejection, but before or on the same date of filing 
an appeal. This declaration could not have been submitted earlier, before the Final Rejection, 
because the reference that is being removed (Peiletier) has been first cited by the Examiner in the 
Final Rejection. This declaration overcomes all pending claim rejections. Thus, a good and 
sufficient reason exists why the present declaration and other evidence is necessary and was not 
earlier presented. 

Declaration 

7. I am a co-inventor of the subject matter claimed in the present patent application. 

At the time I made the subject invention, I was and presently continue to be an employee 
of FUJI XEROX. 

2 
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Atty. Docket No. CQ10196 
PATENT APPLICATION 

DECLARATION OF PRIOR INVENTION IN THE U.S, TO 
OVERCOME CTTED PATENT OR PUBLICATION (37 C.F.R. § L131) 
U.S. Application No. 09/981,735 

On or about May 31, 2000, 1 described my invention m a FUJI XEROX internal 
document FUJI XEROX (FXPAL) Invention Proposal FXPAL-IP-00-013 dated May 31, 2000 
and June 1, 2000 and entitled Xomputer-assisted meeting capturing system." 

Immediately thereafter, the aforesaid Invention Proposal FXPAHP-00-013 was 
submitted to FUJI XEROX legal department Upon the receipt of the aforesaid Invention 
Proposal FXPAL-IP-00-013 by the FUJI XEROX legal department, my invention was assigned 
docket number IP-FA-0013. 

After the detailed review and approval process at FUJI XEROX, the aforesaid Invention 
Proposal FXPAL-IP-00-0 1 3 describing my invention was sent to the law firm of OUff & 
Berridge^ PLC* for preparation and filing. I am informed that the materials describing my 
invention were sent to by Xerox employee Richard Domingo to Steve Roe of the law firm of 
Oliff & Berridge, PLC, on February 10, 2001, as indicated by the Fax Cover Sheet from Richard 
Domingo of Xerox to Steve Roe attached hereto and time-stamped by the facsimile machine 
February 10, 2001 at 13:55. 

The law firm of Oliff & Berridge, PLC proceeded to prepare a patent application and 
filed a provisional patent application serial number 60/268,908 based on the aforesaid Invention 
Proposal FXPALrIP-00-013 on February 16, 2001. The subject patent application claims benefit 
of priority of the aforesaid provisional patent application serial number 60/268,908. 

I hereby declare that all statements made herein of my own knowledge are true, and that 
all statements made on information and belief are believed to be true; and further that these 

3 
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Atty- Docket No. CQ10196 
PATENT APPLICATION 



DECLARATION OF PRIOR INVENTION IN THE U.S. TO 
OVERCOME CITED PATENT OR PUBLICATION (37 C.F.R> § U31) 
U.S. Application No. 09/981,735 

statements were made with the knowledge that willful false . statements and the like so made are 
punishable by fine or imprisonment, or both, under Section 100.1 of Title 1 8 of the United States 
Code, and that such willful false statements may jeopardize the validity of the application ox any 
patent issued thereon. 



Signatures) 



Iwventor(s) 



Full name of sole or first inventor: * . John Boreczky 





Date Country of Citizenship 

Residence Address: 5?6 PuTTqa* tto^ W CffiM^ Of **r?T 
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RECEIVED 

CENTRAL PAX CENTER 

SEP 2 5 2006 

|p F* 0 013 

FXPALlnventlon Proposal FXPAL-l P-OCMJ1 3 



To: Xerox Paicoi Department 
3333 Coyote Hill Road 
Palo Alio, CA 94304 

Proposal Submitted By 



From: FX Palo Alto Labontfury, IfiC- 
3400 Bfllvfcw Ave, Building 4 

P*lo Alto, CA 94304 



Shlngo Uchihfi&hi 
PAL 0049 

sNngoOpal.JtoroxXom 
(630) 013-3907 

Mvutjjer 



John Boreczky 
PAL 0027 

johrtbe patxwox.com 

(090) 013-7303 



Jonathan Foot© 
PAL 0046 

rooted pal Jieioxxofn 

(650) Bld-7071 



Lynn Wilcox 
PAL 0020 

w»COX«pol.z«fttz^om 
(800)813-7974 

Proposml THim 

Computer-Assisted Meeting Capturing System 

&i*f Description 

TO* invention proposal desofbes a system which helps user* capti*© meeting events aa mwlrim a- 
dia docwrawts- It provides intnrtive interface* for camera controls* visual irpresenijtiQii of room 
sctivi&e* end the system statos, flm* lemnj users mate bote* flecfetaas on directing room earners 
and selection of camera. 

Capturing meetinga and preaeuttiknis as multoodi* documents provides varioua opportimibea. A 
captured presenttiian can b» broadcast, allowing remote viewers to participate and share the expe- 
rience with local viewers A video recorded meeting allows people to review & meeting feat they 
have MTPndVd or 10 catch upon a meeting thai fbey have mteted. 

mmesscd md Understood — ^) IjixK^ 
1. SolmJtaer Signature. 

% Snbrahtcx Signature _ 
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The issue i& that it is difficult to make a reasonable Ojuality video of a meeting or a presentation. 
TT» significsajit events have to be captured properly by putting focus on them to order to make the 
video interesting to watch. 

Usually. muWpln cameras are used to capons many aspects of activities in various kind of meet- 
ings. Each camera is able Id pan, tilt and zoom so thai It can bo directed to capture an interesting 
event with high resolution. Multiple video sown from the cameras are fed to a switcher and the 
roost interesting one is selected at each moment daring the meeting The n imiTrmlnn oco of An cam- 
eras and cbe switcher require a lot of know-how and therefore hreouireB a skilled operator to gen- 
erate a recording of a meeting. 

The skill required to captme a meeting causes a mrafcer of problems, k expensive to hire a 
skilled person, and small organisations may not be able to afford one. The person may not be 
available all the time, the operator may be out of the office, or rhere may be more than oriepseaen- 
tatioo at the same time. Also a njctrtcted meeting may not allow any operators to attend- Thus, it w 
dfrSiraNf' ft> malr* It pftg*»Mft tq nrarrl meerings without dedicated opcrncnrs. 

'Videos shot by non-experts are not uccicsung to watch for most of the tuna. A lot of people just 
put a single camera in a zneeting room. This method often tails co capture the wgnificaut events- 
without someone driving it Other people use multiple cameras far recording, but novice users can- 
not handle *ttf»m and tend to leave a single video stream recorded for a long time making the 
resulted video boring. 



Tb produce a uxfal video record of a meeting, two major tasks need to be accomplished, manipu- 
lation of cameras and video stream switching. According all the mams reduce* m*» burden of 
switching but make* camera control mone difficult. When recording only one flfteam. abrupt cam- 
era transition is acceptable as long as a video stream from the camera is not included in the Anal 
record. Ii will be less forgiving if everything is kept as a record. 

Currently, human decisions and stalls ate required to handle (he above two tasks. Some auRiutowd 
solutions nsay be avaUaWe If c 

a number of speakers or restricting the area a speaker can movn* However, those restrictions are 
not applicable in general situations. 

Instead of jumping into bttDdtag an aummated lysiem with limited capaWJIry, our approach is to 
provide enough assistance to novices so that tfany can produce video recordings a* well as skilled 
operators. Participants to meetings or presentation* am not restricted in any way, am i they do no t 
need to wear any uacnmf onabte equipment. Even though the system is not a roily automated meet- 
ing rapturing system, we believe this Is a good strp toward the goal. 



Witnessed and l/nelerrmd by 



^ Vim. 
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System Description 
Overview 
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Figure 1 : System Overview for Camera Controlling Components. 



Oar syvzm consists of three major coinponenls as shown in Figura 1 and 2. A control unit allows 
tamiitive camera control and video stream swMiirtg tm 

display* Images from room CaitusiBa and other image sources as wejl aa soma of the cameras, 
activities in the coim^ence room and n u t iGcaik w* from the system. Tbe intelligent camera con- 
trotter interprets command* from tbe user interlace and cojotrota the cameras. H» controller may 
tscviyc high level corrirrifliKis for ewwomom control of the cameras, such as tracking a person and 
automatic zoom ao^wanenk third component is various analyzers that are scattered over me 
system. Each analyzer baa a small task and sends a remit to tbe control unh to be displayed. 
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switcher status swttcWng control 

Figure 2 ? Syrtem Overview for Video Capturing. 



Control Unit 

fignrc 3 Is a typical view of a control unit. It has a room vjew, and a monitor for video stream*. 
Users cad control ihe camerai mring eitter of them using a pen or odier pointing device sdcd as a 
rnous*. A room view provides an easy way to directly specify «. loading where a nser wsota a OU»- 
era to be aiming "i white a video stream inoautor if suitable foe tending increme ntal ms troctiona to 
ihe cameras. The two interfaces complement eaco ocner co allow for roll camera control. 
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F%ni*3 £ Atypical View of a Control UmL 

Using th& room vt&w 

The? roam view sho ws a floor plan of the cooffcrtak* room as shown irt Rguro 4 . ^p^rccHcfc- 
mg qd any teamm of the floor plan will &end out a command to ihe midligem camera controller 
to drive a camera to mm at toe corresponding locauoa m the actual room A putt-down menu is 
o9cd to provide height information that is missing in (he floor plan mucsentation. Users w select 
ei»e c* me presets tatomeim to 

Igt a 200m paiaiuelci. The zaotn parameter U specified by term* thai era widely used mnong peo- 
ple in broadcast business and afeo easy to understand by others Example of the tttms are "head", 
"shoulder", or "chest*, each texm means to shoot a ckac-up shot of head shiwHa-tm, and chest-- 
up rcrocctivcJy- An adrnntaae of using the tnmw ia that wen can rehrtivdy specify the zoom 
piiamewftottwiiry^ 
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zooming preset 



height value- 



camera selection 
radio bartons *~ 




plan 



Figure 4 3 Room View. 

"track a person". 

The following is the stcp-by-fitep procedure for camera control described above, 

.1. Select jcamnffl. 

2. Select a loom parameter, 

3. Tap or dick on the room vic^r. 

Various methods arc possible far camera selection io stsp I sod 200m pnra mrtrr selection in step 
7. Pull-down menu and radio buttons arc tome of tbe possible unplcmcrttadop candidate*, if a user 
slops step 1 and step 2, the-cbokesprevioufiry selected or default wring? wiB be used. It i» also 
possible to have the system sotomaticany select a camera and room ©erring based on the location 
selected and tbe system'* knowledge of the room Utyoic 

Another way to manipulate the cameras using the room view is to draw a circle oft U. The size and 
the location 0* the chela drawn by a gesture is «nUt m rh* intelligent camera controtlez. Then the 
information is interpreted and generate commands for a camera to shoot the area specified &y the 

circle. 

The following ia the step-by ^atep procedure for camera control using the circle gesture. 
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L Select a camera. 

2- Draw a circle on the room view. 

Step 1 may be omitted a* < H $r tnrc nd before. 
Interacting with the stream monitor 

Die stream monitor is depicted in figure 5. Users can watch all the video streams sent to a 
switcher using the stream monitor. The video stream from the camera elected for controlling is 
hinghted hy a wait© ft™. The vide* <rtream th* b elected by the switcher is h&igbted by a col- 
ored frame id me monitor. The color of the colored frame changes gr^Aially to indicate a duration 
that the rtream has b«m«fcc^ 

red- Color* with low boo value tadicates that the stream Is iKKyetVe^lc^cnmngbBiidnieoiir^ 
users to hold it more* The red frame strongly suggests users to switch to other streams. The transi- 
tion of the frame color is shown in rngme 6. Radio buttons located by the stream monitor ace used 
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Figure 6 : Frame Color KVwiahioo. 

io owitch among streams, A rimer win work as well for. oqjoicnced produoeni to show the 4m- 
lion of the same shot instead of colored flames. 

The stream monitor can be Abo n«d for controlling camera*. Several gestures oa^ fenage of » 
stream allow user? to manipulate the associated camera* Toe commands arc dove-op, dme-nght* 
drive-left, drive-down, anve-op-right, drlvc^p-lcft* tove^towiirrigbt, drlvwtewn-lrft. zoom-m 
and zuuui-OiIti 

A gastare of a stiaJ^ line otlvea a can^ 

line H used to oVtermhw amount of the camera movement. Tapping or clicking on edges of the 
image can be used to drive the associated camef* to a sd^ 
cocked location from the center of s fraroc i» used* 

Oto gesteirt cm also be use^ 

used. An forward spiral gewnre sends a camera to zoom in. while an ontward spiral gesture tolLi a 
camcni to 200m out. Users can specify ft iegi<m of imexost by making a circle gesture around Quit 
aieiLTtegesareisiniw^ 
die erf-fi 

intelligent camera controller 

Too role of the inteffigent camera comroUer is to interpret bigo-level commands from the control 
unit and to generate tew-level commands to drive the cameras, lb reafcee this feare, tbc coattol- 
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captured image 



1 ^ * 




Figure 7 : Cwncra Cflditiiiiaxkjn. 
ler holds geometric information of (he conference room aa well as parameters id drive the mom 
cameras. 

fox a pan/bit capable ca^ym* tbc ueu i e r of rotation (xp, yp, eg) can bo defined getwndik»Hy> Giwa 
that ptuauacm '* id direct the camera to canoed angles arc known, tbe camera can be driven any 
(fireoKQ, (6, to aim at any point in a room within ta range of motion, a is an angle around 
axte, p iff an angle from ttw x-y ptune. Also, a aoom capable camera takes a parameter to control a 
focal length/. By presiding an appropriate parameter, me camera ceo captnrc pictures of any view 
angle. Typically, a pan/tilttsoom capable camera »fces three variables v w v p v, each to specify 
amounts of panning, along and zooming respectively, tbe mappings between the wtabSe* and tbe 
actnal camera parameter* can be described as m equation (1)0X31 If the inappings are linear, 
equation (1X2X3) are rewritten as cqnaooa (4), where c^, ct^ ct^ fJp P,. aro canxra-terxadent 
coDstinii numbers. 
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♦ = V,<v,) 



( 



a„ 0 0 
0 a, 0 



(2) 
(3) 

(4) 



A command from the room view of the control unit coksjSu of three infc ™ w ^^ l^ 3 ^ 
hebtn. Mid * vtew angle Infimnanou. If the caaao^vsfii)er^\v *'^^t^ a ^^ lW 

smiooriaie pre-defined value d to replace the abtfiatt iiaamtt^^^vna^^^^^ 
STft. aflTpf th.ciretei.i^dteABeditelfeteMidB oaring gcawe «ad on^^P^ 
abstract height value to t]ve cOTtroHen Thks is also replaced by an appropriate pio-defined valoe A 
by the controller. 

After replacing an Ac abstract vakies with real values, the ecntrolkf has a location (k. 

tt^ W awl » are. «tu* nw« be covc«d (dX Wnh th. xetue. and »> P^^"^ 

tertocmmwtlie vmbWo f_ v.toddvethespe^ 

aw. This procedure is done in two steps, first, deaainliie ^ a\«id/fwmi the point. C^JO. ^ 
and (*, * tt a* equation (5>T6)<7)- And then tato inverses of eqoatVm <1)(Z)0) to get V v„ V 



6 =» mtan' 



d> = assn 



J 



(6) 



. (7) 

' ~ <* . 

Preset vatue. u«d to replace ateti^t values (jiven by the centre! unix^* T^ll^te***™ 
elS^tThe internet controller adjusts wtoaoinowrjr » wet the otfebtal 

v!»U ft™. end. as motion, edges. **» color, or ""^^^^^^JX^ 
detected, the controller stops adjusting the can>er» portpon. ^ J*™^~^ 1 ^™ 

specifkil by a command. 

Once the ad***** ia done, die ^^''ll^ ^ A ^ a l Cf ^^ B 

^^*e\Zcti«^ 



Yfttrxxud and Understood by 

1. Submitter 51 

2. Sbbmrnsr Signature, 

3. Submitter Signature 




Dai*. 




PAGE 1807 • RCVD AT 9/2512006 6:42:53 PM [Eastern Daylight rone] * SVR:USPT0-EFXRF-5/1 ' DNIS:2738300* CSID:6506258110 * DURATION (mm*s):0M6 



09/25/2006 15:43 6506258110 



SUGHRUE MION 



PAGE 19/37 



autonomously track the person. This tracldng feature can be tamed on and off by a command from 
the control unit. 

IrttalBgent Sensors 

A sensor associated with an intelligent component that analyzes Signal coming ont fium the sensor 
form an intelligent sensor. Various intelligent sensors can bo integrated into mc syncm to support 
users making decision for directing canwra* and switeJbdag video streams. The outputs from me 
ioaslUa^ sensors are visoaDyp^ 

A microphooc array Is one example of sensor*. Multiple ntferopboaes installed m a conference 
room can be used to locate where ttie speaker ia. The oopDitrol unit shows the identified location in 
the mom view by potting a colored blob at the cotrespOndnig place. A user can tap on (ho blob or 
draw a circle around the blob to drive one of the room camera* to capture the speaker 

Activities in a. room can be visually captured tafilng a wide-view-angle camera. IP-99-009 
describes one example. A place with me most intensive motions can be easily located by simply 
taking a difference between every two frames of video from tfcft camera. Such a place is a good 
candidate of an interesting event The system dejects morion activities in the room, and display tbe 
result on the room view of die control unit as colored blobs. Different colors may be used to mdi- 
cava different degrees trf activities. This rerraentotian allows user* to easily identify when* the 
activities aie happening. 

NovaSty 

TbiB invention propose* a system which provides enough assistance for novice user* to produce 
video recordings of meetings and presentations dot are interesting to watch* The a s ntun ce is 
fnr pUmPtiii^ Mt two dements. One element is hUoioWc user interfaces for Controlling room cam- 
«as and a switcher to select a video ttream to be recorded. The otter is automatic nodficarioos of 
the room activities and system status to the users. 

User Interfaces . 

-Autonomous Bcjustment of camera positions allows nsers to select a region of interest with an 
abstract lepresensatiOA, 

■Use of a floor plan for intuitive and direct camera controls. 

"The control unit ojfev an interface for both direct and iocroDPajtal camera control 

Automatic notifications 

•Status of a jwitcbex is notified visually to encourage stream switching at an appropriate fre- 
quency. 

Witnessed and U*dmtood by 
L Submitter 
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•Visual feedback of activities in a conference room is used as a quick overview. 

Ratetod invention proposals, patents, publfcstfons or commercial work 
Other systems have been proposed for Automatic recording of prcsennulans. Blandd presented an 
ftOfy automated system for tetevtefeg and/or recording a presentation!?]. Mukhopadhyay, ct at 
proposed a aimflar system for aiittraatm kcmre caponing system [6). Thewc tystnm combines two 
video streams, a clwc-up of tfa* lecturer and a slide «w, after the tectum. The both tystems work* 
only under limited condinora; the application ia limited to a u tcse n tat io n given by a tingle speaker. 
Instead of building a system with Hrniled use, our approach is to provide enough assistance so thai 
novice users can record any typo of pxweartationa or meetings, thus elinrinatroB iwdo for a dedi- 



FXPAL IP-99-008 and [3] describe user interfaces for controlling cameras and selecting views 
baaed on having a single panoramic camera view. Providing intuitive iTtfnfam is important for 
non-expert*. Bernier et aL d escri be s an automatic camera zoomiog 4y$tem using a face recognition 
technique. This allows users not to go into details for adjustine cameras* Huang et ai reportBd an 
effort to build an automated camera control system[5]. Although their syvtero ia still In its early 
stage and no clear path to a complcto system is shown at Un* moment, their use of a wjdp-view 
camera to observe acnvitwa in a room may be suitable in thii invention too. 

Our ct al. tii c AcjUcd a »Y*te*n wbteh broadcasts a lecture uriog maaWple video stream*. Users can 
rnake a decision to picjc upmost interest 

tcria for the video stream selection. It ia con ce rne d whether the output video is interesting or infar- 
matrve for general viewers. 

1 BernittT, 0.> CoUobert, Femid, IL, Lemaire, V.. Viatel J. E. and CoJtobert* "MUL- ^ 
TRAKiASyt^ttxAntan^hhAtipmon Localization and TVnetcmg In Real-lnne," in 
Proc XCD? '98, pp. 136-139, 1998. 

2 Bianchi M., "AutoAudStonun; a Fully Autotnaric, Multi-Camera System to Televise 
Auditorium FWartBtioo,** Joint DAKPA/NI5T Smart Spaces Technology Workshop, 
GaihVttsbnrg, MD, Jury 199ft. hltp^rVwv#j\ntDAndftor^ 

3 Chio, P.. KajniEkar, A., Itetaneier, S. and Wilcox, ^oteLook: Taking Notes m Meet- 
ings wttb Digital Vfeo Video and mkV* m Proc ACM Multimedia '99. pp. 149-158, 1999. / 

4 Crux, G and HOI R., 'X^potrrog and Playing Mnltirnedla Events widt STREAMS.** in ^ 
Proc ACM Muhhnedla '94. pp. 193-20a 1994. 

5 Huang, Q.. Qn\ Y and Samaxasekera. S., Content Based Active Video Data A^stton 
Via Automated CameranwBU* in Proc ICTP'98, pp. 808-S12, 1998. 

6 Mukbopadhyay, S. and Srnim, "Pawive Capture and Stmcturing Of I^Ctnres/* in Proc. , 
ACM Multimedia '99, pp. 477-487, 1999. 
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Hp* invention been built, mod*, run, or tested? 

The invitation has been partially irapteoidittd ami tested for e*perijne«tal use 

thm Invention usod in * current produces) or pionntd for o» fri a fltiwre 
product(o)? 

Not at this time. 

Datoa of any ptmvtaua or planned future ttoetosure* txtenwt to Xerox 

None 

Source of ouUifr funding 
None 
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Please call to confirm receipt — 

Telephone Number: : 

Xerox Telecopier Fax Number (650) 812-4374 

* 

iMPORTANT/CONRDEKnAL: Thte message is Intended only for the use of me Iretvtdua] or entity 
to which it Is addressed. This message contains information from the Office of General counsel of 
Xerox Corporation wrech may bo privileged, confidential and exempt from cflsotaeura wider 
applicable tow which is not waived due to misdeBvery. rfthe reader ot thte message is not the 
Intended recipient, or the employee or agent respansfcfe for delivering the message to me Intended 
recJpwnt you are hereby notified that any dissemination, dstrtbuoon or copying of this 
communicatioa is strictly prohibited. If you have received this communication in error, please notify 
ua immediately at the telephone number set forth above. We be happy to arrange for the 
return of this message via the United States Postal Services to us at no cost to you. 



Xerox Corporation 
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3333 Coyote Hid Road 
KtfQAIfoCA 94304 
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PAL 0097 

JohnM|nlJwrQ)(.com 

(690) B13-755S 



PAL 0046 

fboteO pal jeeroxxom 

(650)613-7071 



Lynn Wiiecot 
PAL 0020 

wllcox Gpal.xerox*corn 
(600)613-7974 

Proposal THto 

Computer-Assisted Meeting Capturing System 

Brief Dvecrtptiori 

This mveation proposal 

dia docament* it provides fetnlttve interface* far camera controls, wwl repftaeiuadon of room 
activities and the system stara, 0m letting users make better decisions on directing room cameras 
and selection of camera. 

Introduction 

Capturing meeting* and prtjentationi at multimedia documents provided various opportgniriefi. A 
captmcd frocataiion can bo twoadcast, allowing remote vkwera to participate and share th« .xj*- 
fiance wttn local viewers. A vtoeo recorded meeting allows people to review a. meeting mat tbey 
have attended or to catch op on n meeting that tbey have missed. 

Htei»/a»/ Understood by^^^A-^P. 
1- SabmrCDSf 
2. Sabndmx 3i£namn> M 
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The issue is that it i$ difficult to make a reasonable onalily video of ft meeting or a presentation, 
the significant events have to be captured property by potting focus on them ro order to make die 
video interesting to watch. 

n <nan y, multiple cameras are used to capture many aspects of activities in various kind of meet- 
ingv. Each camera Is able Co pan, ttkt and zoom to that ft can be directed to capture an interesting 
event with high rewtoion. Multiple video streams from the ctutrrai ere fed to a switcher and the 
most interesting one is selected at each moment (taring fbe meeting. The maiupiilati otu of tte cam- 
eras and the switid^ require a lc< of lo>>w^ operator to gen- 
erate a recording of a m eet ing. 

The skill required to capture a meeting causes a number of problems, ft is expensive to hire a 
skilled person, and small or gen Unions may not be able to afford one The person may not bo 
available all me rime, the operator may be out of the office, or (here may be more than one presen- 
tation ai the same rime* Also a restricted meeting may not *IU»w any operators to attend. Thus, it is 
desirable to mate it possible to record nicctmgs without dedicated operators. 

Videos shot by nonexperts ate not interesting to watch for roost of the time. A lot of people jnst 
put a (ingle camera in a meeting room This method often fails to capture me significant event? 
wimom soiJKonc driving U. Oow peop^ 

not handle mem and tend to leave a single video stream recorded for a kmg time making the 
resulted video boring. 

To pitidijce a usefbJ video reco^ 

larion of cameras and video strewn switching. Recording all the stream* reduce* to burd en of 
switching but makes camera concrol more difficnk 

era transmort is acceptable as long u a video stream from the earner is not mended in me final 
record. It wiK he less forgiving if everything i* kept a* a record. 

Currently, human dedskxns and skills are required to handle the above two tasks. Some atitomatod 
trolnthma i w y br ftrnflftH* f* condH""* «™M *»*^ *-e- fiamg a typn of a meeting, limiting 
a number of speakers or restricting the area a speaker can move. However; those restrictions are 
not applicable in general meeting situations. 

Instead of jumping into building an auxornared system with limited capability, our approach is to 
provkfc enough aWtance to novices so mat tfaey cms produce video i™tningfl es weU as skilled 
operators. Participants m meetings or pricsenwiooa are nor restricted in any way* mid they do no t 
need to we» say uiicomfo^ 

big capturing system, we believe mis is a good stop toward the goat 
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System Description 
Overview 



Additional Sensors 



V7 



actional info 



camera 
status 



Control 
Una 



analyst rosu* 



control tngpial 



intelligent 
Camera Comralter 




camera status 



Room 



Other QQunsee 



control algnaj 



1/ 



SouruB 
Arwyzor 



Figure 1 1 System Overview for Camera Controlling Components. 



Oar system cons to of ftrem^OT 1 and 2. A control upit allows 

mtylffvc camera control and video stream swathing using a graphical hHierttee^Tbofau^;*** 
displays ivon&A from room camera and other image sources as well ax aatna of mo cam eras, 
activities in tbe conference mom and notifications fiom. me system. Tbe inteUiynt earner* con- 
troDer Uwerprctt commands from tfce user interface and controls the cameras. The cog troger m ay 

automatic zoom adjustment Tbe mfed component b various analyieni mac ate acaiteiedovci tte 
sy ansa. Each analyzer baa a amidl task and sends a result to die control unit to be displayed. 



Witnessed and Understood by . 

2LSnbmmerSlgnamre jf^^gij^/^ ^ Dote V/dffio 

3* Suboihier Signature — 

JL.AAWlAxVT^fo- Iter £///g>0- 



PAGE 25/37 ' RCVD AT 9125/2006 6:42:53 PM [Eastern Daylight Time] * SVR:USPT0-EFXRF-5!1 » DNIS:2738300 * CSID:6506258110 ' DURATION (mm-ss):07-16 



09/25/2006 15:43 . 6506258110 SUGHRUE MION 

FH9- 10-2001 13:56 FRQM:XERO , : PflLD RLTO 650-91^-4574 TO \ 835 27HY 



PAGE 26/37 




*witeftaf status switching control 

Figure 2 ; System Overview far Video Capturing. 



Control Unit 

Figure 3 is a typical view of a control unit. Ik has a room view, and a monitor fox video scream*. 
Users can control the cameras using either of tbem using a pen or after pointing device sucn as a 
nuniM, A room view provides an easy way to dixectiy specify a location where a w wanta a ca*n> 
era to be aitning at, while a video stream monitor ia suitable for sending incremental inatructioai to 
the cameras. The ttro Interfax 



1. Snbmraw Signgturr,,,,^^^ faqjtf^/ — Dat e Sfaf/OO 
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Figure 3 1 A typical View of a Control Unit. 



Using the mom view 

The room view shows a floor plan of me conference roora a« shown in Flguoe 4. Tapping or click- 
ing on any location of the floor plan *iC seod out a command to the intelligent ca mera c<m noDer 
to drive a camera to aim at the cotreapondiDg location hi the actual room A putt-down menu ia 
used to provide height mfonration mat ia mianng In the floor plan r epKOjert H trfr iL User* can select - 
one of the presets in foe menu hie ta stmdm^or M »eafc^.AiiDte 

set a room parameter. The zoom pwvnfaer la specified *y terms mat are widely used anK?n 8P l =£' 
pie in broadcast business and also easy to aodrrstand by omen. Example of the terms are "heaor. 
"sbouMer". or "chesT, each term means to short a d08**iap shot of bead, shoulder-up. and chest- 
up respectively. An advantage of using me ttroia i* that user* can relanvtry apecify the zoom 
parameter not worrying about adjusting it Other mf onDatiOtt i»ay he sent to the cctttroUer aoch as 
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zoaQUQg preset 



height vahit- 



camera selection 
radio buttons ' 




Ftggrv 4 r Room View. 

"tntclc o person". 

HiefoUowing to cbc step-by-rtep procedure for camera control described above 

1. Select a camera* 

2. Select A zoom parameter. 

3. Dip or click on the roam view. 

Various ipetbads are possible far canxTasdccbtn) in step 1 and zoctd parameter election in step 
XPtdWo^n^n and radio butt^ 

skips step 1 and step 2, the choices previously selected or default settings will be csed. U Is also 
possible to haw the system automatically select a camera and zoom seeing based on toe location 
sejpeted and the gygtmn's knowledge of tbe room layout 

AnolheT way io nuiupiito 

ma facation of ttje cizclo drawn by a gesture is sent to the intelligent camera contzolte Then me 
information u interpreted and generate commands for a camera to shoot the area specified by che 
circle. 



The following is the step-by-step procedure for camera control using ttae. circle gesture. 
Witnessed <md Understood by^^^L^P l foe* 
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Figure 5 : Stream Monitor. 



1. Select a canm 

2. Draw a circle on die mom view. 

Seep 1 may be omitted as discussed before. 
Interacting with the M monitor 

The stream monitor is depicted is Ffrxe 5. Usees can watch all ate video s&eams sent to a 
ditcher ossng the sueam monitor. The video stream from the camera selected for controlling v 
hiHgbtedbyn white fran^Tbev^ 
aredfrao^hithemoiutw.TD^ 

tfwt the stream to oeefl «Jec«L The cote goes from gw to U«, blue to yellow, and finally to 
red. Colors wift low hoe vatoe indicates thai the stream is not yet kept long eno qghan d encourage 
users m hold U more. Tlie red ftaitm Stten^ 

two of die frame color is shown in figure 6. JUdio buttons located by the steam monitor ate used 
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Rpireti : fi^mc Color TVaasdrJon. 

10 switeh among stream*. A timer will work as well far experienced producer! to ehow the dm- 
ttan of tbe flame dust instead of colored franca, 



The stream TDorritor can be also used for controlling caunera*. Several gestures on an image of a 
stream allow users to manipulate the Associated camera. The commands are drive-iij, drive-right, 
oVive-Ieft, drive-down, (tove-op-right, dtWe-np-left, drive^lowiwigbt, drive>down-left, xoom-iD 
and zoom-out. 

A gesrqr© of a straight tine drrra a cameia to the direction that die line is drawn. The length of did 
tine is used to fettnnine amount of the camera movement. Tapping or clicking on edge* tf* 6 
image can be used Co drive me aatnriatrr! camera fer a small degree. Tbe direction of the tapped or 
cocked location from the center, of a rrarae ia used. 

Olber gesture* can also be used. For example the zooming gestures ftom FXPAL can be 
wxL An inward spiral gesture sanda a camera to zoom in, while an outward spiral gesture tells a 
Ljii Mg i it tp zoom out. Users can specify a region of Jnteresi by making a circle gusrure around that 
area. The gesarrc is irUtjpreted by me rmglugeni camera controller and a camera is driven to focng 
the area* 



intelligent camera controller 

Tbe role of the imcHigem <i**r**<* controller is to imuutul bigb-tevtl commands from die control 
unit and to generate krw-lc vd c^rnriisiKi* to oWx thecal 
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captured frnag* 



Figure 7 i Camera Coccdmatkon, 
ter holds gcQTPecrte inl6miad<m of the conference room as well as parameters to drive (be room 
cameras. 

For a pan/tilt capable amen, the center of rotation C?o» £q) can tic defined gAomefricoUy* Given 
that parameters to direct the camera to desired angles ae known, die camera can be chiven any 
dirouon (0, ♦) to aim at any point in a room within its range of motioa 9 Ifl an angle around ^ 
axis, * is an angle from tbe JC^y plane, Alao, a zoom capable camera takea A pwn^ «» control a 
focal length/ By providing an appropriate 

angle. Typically, a panftiltftooni capable caroeni takes three variables v, each to specify 
amounts of panning, tftaog aad zooming respectively. Tbcfnappinga between the variables andtbe 
actual camera pafameten eas be described as m equation (WC2X3). If the mapping* a re Hnear^ 
equation (t)(2X*) are rewritten as equation (4X where 0^,. cv o^s P> Rr 8 " 5 camera-dependent 
constant nwn&eri, 



(i) 



WEme.ssc<* <wtd Understood by d 

u Submitter Sigtwrnre^ 
2. Submitter Signature.. 




% Submitter Signature, 



PAGE 31/37 • RCVD AT 9Q5I2006 6:42:53 PM [Eastern Daylight Time] * SVR:USPT0-EFXRF-5/1 » DNIS:2738300 * CSID:6506258110 1 DURATION (mm-ss):0M6 



09/25/2006 15:43 650G258110 



SUGHRUE MION 



PAGE 32/37 



FEB^0-a001 13:59 FRDM:XERGX . ' •■■JO AUTO 650-812-4274 TO' f S7B7 P.011'014 



/ = v/v 









0 o) 


IS] 




Pp 






0 








P, 








0 Of, 









0) 
Of) 

<4> 



A command from the room view of the control unit consists of ttat» informal *-y location 
heiahr. and a vWsw angle information. If the command Is genemfctd by a tapping gestnre, the view 
ScW annation *£v*o in an abstr** f an* such tt <T**" or ^btsi". Tbc~i«oI^an 

ab^o^ bright valuator 
by the cavtioV^. 

Aton^lacfosaniheab^ 

toaiin « ^ » ^ttai mimbB ewvorft^^ 

l^detemnaestheva^ 

area. Ute procedure is done in two steps, Rm, tooroino ©, and /from the points C*fo 
and Cc, * A) » equation (5)(6)0* And then utte Jnvenes of cqvfikm (1X2*3) to gel V v p v r 



0 = atao 



$ m ASttV 




' / ! 3 



(5) 



(6) 



/ * —3 — 

p^Taluc* medio replace abtn^ 

estimation. The totm&nt control adjust* antonomocsly to me* the oriajnal 

varknis feature*, such as roodon, «tfi=*> dan co^ ct ^bm^^ o( Bbcye^^ * 
deleted, the control stops a^tia* the camera pwirion. The ^^^^^ 

specified by ft command 

adjustiiignVdi^cmof tbecanm images, the camera can 
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autowxniousry track the person. This {racking feature can be tamed on and off by a conunasd from 
the control unit. 

IntelMgent Sensors 

A sensor associated with an intelligent component mat analyzes signal coming oat from the sensor 
form an intelligent aenftor* Various intelligent sensors can be integrated into me system to support 
users making decision for directing cameras and swta&rog video stnnnjis. Tbe oinpoia Irani me 
intelligent tensors arc visnaDy present in the control unit so that users can easily recognize them. 

A microphone array Is one example of sensors. Multiple microphones installed in a corofeitoc* 
room can be used to locate where the speaker is. The control unit shows the ide nt ified location in 
tbe room view by putting a colored blob at tbe cogeaponding place. A user can tap on the blob or 
draw a circle around the blob to drive one of the room cameras to capture the spc*kfX 

Actrvities in a room can be visually captured using a wide- view-angle camera. HMKMXtt 
describes one example. A place with me most intensive motions can be easily located by simply 
taking a difference between every two frames of video from the camera. Such a place is a good 
candidate of an interesting event. The system detects motion activities in the room, and display tbe 
result on the room view of die control unit as colored blob*. Different color* may be used to indi- 
cate different degrees of activities. This representation allows users to easfly identity wfcerc die 
activities are happening. 

This invention proposes a system which provides enough assistance for novice usbzy to produce 
video recordings of meetings and presentations that ate interesting to watch, Tbe assistance is 
innjfemcnted as two elements. One element is intuitive user interfaces foe "mtrofling room cam- 
eras and a switcher to select a video stream to be recorded. The other is automatic notificariritifl of 
the room activities and system status to the users. 

User Interlaces . 

•Autonomous adjuster*** of camera positions allows osers to s elect a region of interest with an 
Abstract representation. 

♦Use of a floor plan for innritive and direct camera controls. 

•The control unit afifora an mterface for both direct and mcremental carom control 

Automatic notfflcations 

-Status of a switcher is notified visually to en co u ra ge stream switching at an a^h optic te fre- 
quency. 
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•Visual feedback of activities in a conference room is trod as a qmck overview. 
Related Invention proposals, patents, publications or commercial work 

Other systems have been, proposed for automatic recording of yw«** r » ri ™*« Bimxcm' pmuried mi 
fuliy automated aysem for (devising and/or recording a presentationp}. Miishopadhyay, et al. 
proposed a similar system fur automatic Vetera capturing system [6]* Their system combines two 
video streams, a close-op of fee lecturer and a slide view, after the lecture. The both systems works 
only under limited candirions; the application ia limited to a Pfcscntadoti given by a sfaglc speaker 
Instead of boOding a system with limited use; our approach is to provide enough assferance 60 dwtt 
novice users can record any rype of presentations or meetings, thus eltmlnarmg seeds for a dedi- 
cated opEramc 

FXPAL IP-99-006 and 13} describe user fttfrrftras for controlling cameras and selecting views 
based on having a single panoramic earners view. Providing unuitw interfaces Is important for 
non-expert«- Bcrzricr ct al- describes an automatic camera zooming system using a face recognition 
teebniqa*. TOs allows users not to so into details for adhrtmg cameras. Hoaag at al icported an 
effort to build an automated camera control system(5]. Although their system is still in its early 
stage and no dear path to a complete system is shown attms moment* me*r use of a wide-view 
camera Co observe activities m a room may be suitable in this inventwi Uoo, 

Cruz et ol. presented * system wfaada W*dc«±« a lector* ask* mx*Wp)e video araams. Users can 
make a decision to pick the most interesting 5tream{4]. They solely use personal interest as me cri- 
teria for the video stream selection, ft is concerned wfeexber me outpui video is interesting or infor- 
mative far general viewers. 

1 Bernier, 0. p CoTiobert, M., Feraud, JL, Lemsire, V, Vlalet, J. EL and CofloberU ~MU1-^ 
TRAK: A System for Aatomatfc MnJaperson LocaKtwrtoii and TracJdng m fteal-'DTne,'* m 
Froc iCSP '98, pp. 136-139, 1998. 

2 BiancM, Ki, "AutoAwditoriuo: a Fully Avtomadc, Muni-Camera System to Televise 
Auditorium I~n i n iiiniiin" Joint OARIVWNIST Smart Spaces Technology Workshop, 
Gaithersburg, MD, inly 1998. bttp^Avwwj\ntoAndUorium.c^^ 

3 Olio, P, Kapuskan, A., Reiflneier, 5. and Wilcox* L, "NoteLook: Taking Notes in Meet- 
ings with Digital Vieo Video and Ink," in F*oc ACM Multimedia '99, pp. 149-158, 1999. / 

4 Cruz, n and Hid* IL, 4 *Capturing and Playing Multimedia Events with STREAMS," in • 
Proc ACM Mulmiiedis *94. pp. 193-200, 1994, 

5 Huang, Q*, Cui, Y. and Samarasekera, S., "Content Based Active Video Data Acquisition 
Via Automated Cameramen/* in. froc. IOF98, pp. J108-812,, 1998. 

6 MuldiopadlrYay, S, and Smith, B„ "Passive Capture and Structuring of Lectures." in Proc. , 
ACM Multimedia '99, pp. 47M87, 1999. 
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Has Invention been built, made, run, or tested? 

The invention has been partially implemented and tested far experimental use. 

Is the Invention used in a current produces) or planned for use in * future 
produces)? 

Not 3tt tfus thna- 

Omm of any previous or planned future disclosure* external to Xerox 

None 

Source of outside funding 

— 
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